Remote work supporting system, remote work supporting method, and program

ABSTRACT

A remote work supporting system shares an image  200  photographed by a wearable terminal between the wearable terminal and a remote terminal in order to support remote work. An image acquiring unit acquires the image  200  photographed by the wearable terminal. A screen sharing unit screen-shares the image  200  between the wearable terminal and the remote terminal. An accepting unit accepts an input of instruction information for the image  200  from the remote terminal. A first display unit superimposes and displays the image  200  and a virtual object  201  corresponding to the instruction information on the wearable terminal. A second display unit superimposes and displays the image  200  and the virtual object  201  on the remote terminal. An image recognizing unit analyzes the image  200  to recognize a state of a body part. A changing unit changes a display of the virtual object  210  by the first display unit and the second display unit based on the recognized state.

TECHNICAL FIELD

The present invention relates to a technique for supporting remote work.

BACKGROUND ART

There is known a technique for sharing a screen among a plurality ofdevices. For example, Patent Document 1 discloses a technique in which aserver acquires screen information of a plurality of clients anddisplays the screen information in a list form.

PRIOR ART DOCUMENT Patent Document

Patent Document 1: Japanese Patent Application Publication No. 2003-6062

SUMMARY OF THE INVENTION Technical Problem

Meanwhile, at a work site of a factory or the like, there is a casewhere an instructor remotely gives a work instruction to a worker for aremote work support. It is preferable that the worker responds to theinstruction of the instructor in a way that both hands are free like avoice, in order to perform the work. However, when a noise level of thework site is high, it is difficult to support the remote work by thevoice.

It is an object of the present invention to realize smooth remote worksupport even when a user of a wearable terminal is in an environmentwhere it is difficult to receive a remote work support by the voice.

Technical Solution

The present invention provides a remote work supporting system forsharing an image photographed by a wearable terminal between thewearable terminal and a remote terminal in order to support remote work,the remote work supporting system including an image acquiring unit thatacquires the image photographed by the wearable terminal, a screensharing unit that screen-shares the image between the wearable terminaland the remote terminal, an accepting unit that accepts an input ofinstruction information for the image from the remote terminal, a firstdisplay unit that superimposes and displays the image and a virtualobject corresponding to the instruction information on the wearableterminal, a second display unit that superimposes and displays the imageand the virtual object on the remote terminal, an image recognizing unitthat analyzes the image to recognize a state of a body part, and achanging unit that changes a display of the virtual object by the firstdisplay unit and the second display unit based on the recognized state.

Effects of the Invention

According to the present invention, even when the user of the wearableterminal is in an environment where it is difficult to receive remotework support by the voice, smooth remote work support can be realized.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an example of a configuration of a remotework supporting system 1.

FIG. 2 is a diagram showing an example of an appearance of a wearableterminal 10.

FIG. 3 is a diagram showing an example of a hardware configuration of awearable terminal 10.

FIG. 4 is a diagram showing an example of a hardware configuration of aremote terminal 20.

FIG. 5 is a diagram showing an example of a hardware configuration of aserver device 30.

FIG. 6 is a diagram showing an example of a functional configuration ofa remote work supporting system 1.

FIG. 7 is a sequence chart showing an operation of a remote worksupporting system 1.

FIG. 8 is a diagram showing an example of a virtual object 210.

FIG. 9 is a diagram showing an example of a screen 220 displayed on adisplay device 16.

FIG. 10 is a diagram showing an example of a screen 230 displayed on adisplay device 26.

FIG. 11 is a diagram showing an example of a change in a display of avirtual object 210 in a display device 26.

FIG. 12 is a diagram showing an example of a screen 240 displayed on adisplay device 16 in a modified example.

FIG. 13 is a diagram showing another example of a screen 250 displayedon a display device 16 in a modified example.

FIG. 14 is a diagram showing an example of a screen 260 displayed on adisplay device 16 in a modified example.

FIG. 15 is a diagram showing an example of a screen 270 displayed on adisplay device 16 in a modified example.

FIG. 16 is a diagram showing an example of a screen 280 displayed on adisplay device 16 in a modified example.

FIG. 17 is a diagram showing an example of a screen 290 displayed on adisplay device 16 in a modified example.

FIG. 18 is a diagram showing an example of a screen 300 displayed on adisplay device 26 in a modified example.

DESCRIPTION OF REFERENCE NUMBERS

1: remote work supporting system, 10: wearable terminal, 20: remoteterminal, 30: server device, 101: image acquiring unit, 102: acceptingunit, 103: voice recognizing unit, 104: object generating unit, 105:screen sharing unit, 106: first display unit, 107: second display unit,108: image recognizing unit, 109: changing unit, 110: responsegenerating unit, 161: display panel

DETAILED DESCRIPTION OF THE EMBODIMENTS Configuration

FIG. 1 is a diagram showing an example of a configuration of a remotework supporting system 1. The remote work supporting system 1 is asystem for supporting remote work. It is noted that the remote worksupporting system 1 is designed for supporting the remote work and isdifferent from, for example, a video conference system for holding aconference while watching a face with a remote person. In the remotework support system 1, a wearable terminal 10 has a function ofphotographing an image, and the image photographed by the wearableterminal 10 is shared between the wearable terminal 10 and a remoteterminal 20. In this case, a user of the remote terminal 20 canunderstand a situation of a user of the wearable terminal 10 by viewingthe image photographed by the wearable terminal 10. For example, aninstructor having a predetermined qualification may use the remoteterminal 20 and an on-site worker may use the wearable terminal 10. Inthis case, the instructor can give a work instruction to the on-siteworker from a remote location. In the following description, the user ofthe wearable terminal 10 is referred to as a “first user” and the userof the remote terminal 20 is referred to as a “second user”.

The remote work supporting system 1 includes the wearable terminal 10,the remote terminal 20, and a server device 30. The wearable terminal10, the remote terminal 20, and the server device 30 are connected via acommunication line 40. The wearable terminal 10 is wirelessly connectedto the communication line 40. The communication line 40 may include, forexample, the Internet and a wireless communication network.

FIG. 2 is a diagram showing an example of an appearance of a wearableterminal 10. In this example, the wearable terminal 10 is a binocularglasses-type terminal to be worn on a head of a first user. The wearableterminal 10 includes a frame 19 and two display panels 161. The frame 19is a member holding the two display panels 161. The display panel 161 isa plate-shaped member having light transparency and is provided in frontof the first user's eyes, i.e., within a field of view of the firstuser. The first user can see a front space through the display panel161. Here, “having light transparency” means that a back side can beseen through the display panel 161. The display panel 161 may betransparent or translucent.

FIG. 3 is a diagram showing an example of a hardware configuration of awearable terminal 10. The wearable terminal 10 includes a processor 11,a memory 12, a communication interface 13, a storage device 14, an inputdevice 15, a display device 16, and a camera 17. These devices areconnected via a bus 18.

The processor 11 loads a program in the memory 12 and executes theprogram, thereby performing various controls or processing. As theprocessor 11, for example, a CPU (central processing unit) may be used.The memory 12 stores the program to be executed by the processor 11. Asthe memory 12, for example, a ROM (read only memory), a RAM (randomaccess memory), or a combination of the ROM and RAM may be used. Thecommunication interface 13 is connected to a communication line 40 andperforms a data communication according to a predetermined wirelesscommunication standard. The storage device 14 stores various data andprograms. As the storage device 14, for example, a flash memory may beused.

The input device 15 is used for an operation of the wearable terminal10. As the input device 15, for example, a key may be used. The displaydevice 16 displays various kinds of information on the above-describeddisplay panels 161. The display device 16 may include, for example, thedisplay panels 161 and a projector that projects an image on the displaypanels 161. When the image is displayed on the display panels 161, theuser can view the image displayed on the display panels 161 togetherwith the front space.

The camera 17 includes an image pickup device and an optical system, andphotographs an image of a subject by forming the image of the subject onthe image pickup device by the optical system. As the camera 17, forexample, a digital camera is used. The camera 17 is installed at aposition and direction capable of photographing the image along a fieldof view of the user.

FIG. 4 is a diagram showing an example of a hardware configuration of aremote terminal 20. The remote terminal 20 may be, for example, apersonal computer. The remote terminal 20 includes a processor 21, amemory 22, a communication interface 23, a storage device 24, an inputdevice 25, a display device 26, and a microphone 27. These devices areconnected via a bus 28.

The processor 21 loads a program in the memory 22 and executes theprogram, thereby performing various controls or processing. As theprocessor 21, for example, a CPU may be used. The memory 22 stores theprogram to be executed by the processor 21. As the memory 22, forexample, a ROM, a RAM, or a combination of the ROM and RAM may be used.The communication interface 23 is connected to a communication line 40and performs a data communication according to a predeterminedcommunication standard. The storage device 24 stores various data andprograms. As the storage device 24, for example, a hard disk, a flashmemory, or a combination of the hard disk and the flash memory may beused.

The input device 25 is used for an operation of the remote terminal 20.As the input device 25, for example, a keyboard and a mouse, a touchsensor forming a touch panel, or a combination of the keyboard, themouse, and the touch sensor may be used. The display device 26 displaysvarious kinds of information. As the display device 26, for example, aliquid crystal display may be used. The microphone 27 collects a voiceand converts it into a voice data.

FIG. 5 is a diagram showing an example of a hardware configuration of aserver device 30. The server device 30 includes a processor 31, a memory32, a communication interface 33, and a storage device 34. These devicesare connected via a bus 35.

The processor 31 loads a program in the memory 32 and executes theprogram, thereby performing various controls or processing. As theprocessor 31, for example, a CPU may be used. The memory 32 stores theprogram to be executed by the processor 31. As the memory 32, forexample, a ROM, a RAM, or a combination of the ROM and RAM may be used.The communication interface 33 is connected to a communication line 40and performs a data communication according to a predeterminedcommunication standard. The storage device 34 stores various data andprograms. As the storage device 34, for example, a hard disk, a flashmemory, or a combination of the hard disk and the flash memory may beused.

FIG. 6 is a diagram showing an example of a functional configuration ofa remote work supporting system 1. The remote work supporting system 1includes an image acquiring unit 101, an accepting unit 102, a voicerecognizing unit 103, an object generating unit 104, a screen sharingunit 105, a first display unit 106, a second display unit 107, an imagerecognizing unit 108, a changing unit 109, and a response generatingunit 110.

In this example, the image acquiring unit 101 and the first display unit106 are provided on the wearable terminal 10. These functions arerealized by a cooperation of a program stored in the memory 12 and theprocessor 11 executing the program. The accepting unit 102 and thesecond display unit 107 are provided on the remote terminal 20. Thesefunctions are realized by a cooperation of a program stored in thememory 22 and the processor 21 executing the program. The voicerecognizing unit 103, the object generating unit 104, the screen sharingunit 105, the image recognizing unit 108, the changing unit 109, and theresponse generating unit 110 are provided on the server device 30. Thesefunctions are realized by a cooperation of a program stored in thememory 32 and the processor 31 which executes the program.

The image acquiring unit 101 acquires an image photographed by thecamera 17. The screen sharing unit 105 screen-shares the image acquiredby the image acquiring unit 101 between the wearable terminal 10 and theremote terminal 20. When the second user instructs this image by avoice, the accepting unit 102 accepts an input of a voice dataindicating the voice collected by the microphone 27. The voicerecognizing unit 103 recognizes the content of the voice of the seconduser by analyzing the voice data accepted by the accepting unit 102. Awell-known voice recognition technique may be used to recognize thecontent of the voice. The object generating unit 104 generates a virtualobject based on the content of the voice recognized by the voicerecognizing unit 103. This virtual object may be, for example, one inwhich the content of the voice is described.

The first display unit 106 displays the image acquired by the imageacquiring unit 111 and the virtual object generated by the objectgenerating unit 104 on the wearable terminal 10 by superimposing them.This display may be performed using an AR (augmented reality)technology. The second display unit 107 displays the image acquired bythe image acquiring unit 111 and the virtual object generated by theobject generating unit 104 on the remote terminal 20 by superimposingthem.

The image recognizing unit 108 analyzes the image acquired by the imageacquiring unit 111 and recognizes a state of a finger of the first user.This state includes a movement, a shape, or a position of the finger.For example, the first user may operate the virtual object using thefinger seen through the display panel 161. This operation may be, forexample, a gesture operation of moving or rotating the virtual object ortouching the virtual object.

The operation on the virtual object is recognized based on the imagephotographed by the image acquiring unit 101 and the virtual objectdisplayed by the first display unit 106. For example, coordinates of theimage photographed by the image acquiring unit 101 are converted so thatthe image becomes an image viewed from a viewpoint position of the firstuser. Then, a composite image into which the virtual object and theconverted image are combined is generated so that a field of view of thefirst user is reproduced. The operation on the virtual object isrecognized based on a positional relationship between the finger of thefirst user and the virtual object in the composite image and the stateof the finger of the first user.

The changing unit 109 changes the display of the virtual objectdisplayed by the first display unit 106 and the second display unit 107based on the state recognized by the image recognizing unit 108. Forexample, the changing unit 109 may change the display of the virtualobject by transmitting control information for changing the display ofthe virtual object. In another example, the changing unit 109 maytransmit an image data representing changed virtual object. The responsegenerating unit 110 generates response information based on the displayof the virtual object and the state of the finger recognized by theimage recognizing unit 108. A relationship among the display of thevirtual object, the state of the finger, and the content of the responseinformation may be predetermined. When the response information isgenerated by the response generating unit 110, the second display unit107 further displays the response information.

In the following description, in a case where the wearable terminal 10is described as a subject of processing, it means that this processingis executed in cooperation with the program stored in the memory 12 andthe processor 11 executing the program. The same applies to the remoteterminal 20 and the server device 30.

Operation

FIG. 7 is a sequence chart showing an operation of a remote worksupporting system 1. Here, it is assumed that a first user is a workerwho works at a work site and a second user is an instructor at a remotelocation. In this case, the worker wears the wearable terminal 10 andthe instructor uses the remote terminal 20 to instruct the worker fromthe remote location. It is noted that the term “remote location” meansthat it is a different place and it is not always necessary to be farapart.

In this case, the worker photographs an image of the work site using acamera 17. This image may be a moving image or continuous still images.The image acquiring unit 111 acquires the image 200 (hereinafterreferred to as a “photographed image 200”) photographed by the camera17. A wearable terminal 10 transmits an image data representing thephotographed image 200 to a server device 30. The server device 30receives the image data from the wearable terminal 10. A screen sharingunit 105 transmits this image data to the wearable terminal 10 and aremote terminal 20. Each of the wearable terminal 10 and the remoteterminal 20 receives the image data from the server device 30. A firstdisplay unit 106 displays the photographed image 200 corresponding tothe image data on a display device 16. In addition, the second displayunit 107 displays the photographed image 200 corresponding to the imagedata on a display device 26. As a result, screen sharing is performedbetween the wearable terminal 10 and the remote terminal 20. Theinstructor understands a work situation of the worker by viewing thephotographed image 200 displayed on the display device 26 and providesthe worker with a work instruction by a voice. When the instructorprovides the work instruction by the voice, processing of step S101 isstarted.

In step S101, a microphone 27 collects the voice of the instructor andconverts the voice into a voice data. Here, it is assumed that the voice“please photograph the right side” is issued by the instructor. In thiscase, this voice is collected and converted into the voice data. Theaccepting unit 102 accepts the converted voice data.

In step S102, the remote terminal 20 transmits the voice data to theserver device 30. The server device 30 receives the voice data from theremote terminal 20.

In step S103, a voice recognizing unit 103 recognizes the content of thevoice by analyzing the voice data and converts the recognized contentinto a character data. In this example, the content of the voice “pleasephotograph the right side” is converted to the character data. Thecharacter data is used as instruction information.

In step S104, an object generating unit 104 generates a virtual object210 based on the instruction information.

FIG. 8 is a diagram showing an example of a virtual object 210. Thevirtual object 210 includes instruction information 211 of “pleasephotograph the right side”. In addition, the virtual object 210describes an arrow 212 directed in an X direction in the figure andresponse information 213 of “YES”. The response information 213 isarranged at a position corresponding to the arrow 212. This indicatesthat the response “YES” is performed by moving the virtual object 210 inthe X direction in the figure. Furthermore, the virtual object 210describes an arrow 214 directed in a -X direction in the figure andresponse information 215 of “NO”. This indicates that the response “NO”is performed by moving the virtual object 210 in the -X direction in thefigure. While the virtual object 210 is a quadrangle in the exampleshown in FIG. 8, other shapes may be used.

In steps S105 and S106, a screen sharing unit 105 transmits an imagedata representing the virtual object 210 generated in step S104 to thewearable terminal 10 and the remote terminal 20. However, theinstruction information of “please photograph the right side” obtainedin the above-described step S103 is transmitted to the remote terminal20 together with the image data. Each of the wearable terminal 10 andthe remote terminal 20 receives the image data from the server device30.

In step S107, the first display unit 106 displays the virtual object 210corresponding to the image data on the display device 16. Morespecifically, the virtual object 210 is displayed on the display panel161.

FIG. 9 is a diagram showing an example of a screen 220 displayed on thedisplay device 16. The screen 220 includes the photographed image 200and the virtual object 210. The virtual object 210 is superimposed anddisplayed on the photographed image 200. As a result, the worker seemsto have the virtual object 210 appear on the actual work site.

In step S108, the second display unit 107 displays the virtual object210 corresponding to the image data received from the server device 30on the display device 26 like the first display unit 106.

FIG. 10 is a diagram showing an example of a screen 230 displayed on thedisplay device 26. The screen 230 has a display area 231 and a talk area232. The display area 231 includes the photographed image 200 and thevirtual object 210. The virtual object 210 is superimposed and displayedon the photographed image 200. Instruction information of “pleasephotograph the right side” is displayed on the talk area 232.

The worker looks at the screen 220 shown in FIG. 9 and responds to theinstruction of the instructor. In the example shown in FIG. 9, a gesturefor moving the virtual object 210 with a finger in the direction of thearrow X in the figure is associated with a response “YES”. Further, agesture for moving the virtual object 210 with the finger in thedirection of the arrow -X in the figure is associated with a response“NO”. In other words, the response “YES” is predetermined for a movingdirection in the direction of the arrow X in the figure, and theresponse “NO” is predetermined for a moving direction in the directionof the arrow -X in the figure.

For example, in a case of responding to the instruction of theinstructor as “YES”, the worker, in front of the camera 17, performs thegesture for moving the virtual object 210 with the finger in thedirection of the arrow X as shown in FIG. 9. More specifically, theworker moves the finger in the direction of the arrow X in the figurefrom a position having a predetermined relationship with the virtualobject 210. The position corresponding to the virtual object 210 is, forexample, a position overlapping with the virtual object 210 in the fieldof view of the worker.

In step S109, an image including the finger of the worker isphotographed by the camera 17. An image acquiring unit 101 acquires theimage photographed by the camera 17.

In step S110, the wearable terminal 10 transmits an image datarepresenting this image to the server device 30. The server device 30receives the image data from the wearable terminal 10.

In step S111, an image recognizing unit 108 recognizes the gestureperformed by the worker by analyzing the image data. In this example,the gesture for moving the virtual object 210 with the finger in thedirection of the arrow X in the figure is recognized. More specifically,the movement of the finger moving in the direction of the arrow X in thefigure from the position having the predetermined relationship with thevirtual object 210 is recognized.

In steps S112 and S113, a changing unit 109 transmits, to each of thewearable terminal 10 and the remote terminal 20, control information forchanging the display of the virtual object 210 by the display device 16and the display device 26, based on a state recognized in step S111. Thewearable terminal 10 and the remote terminal 20 each receive the controlinformation from the server device 30.

In steps S114 and S115, the first display unit 106 and the seconddisplay unit 107 each changes the display of the virtual object 210 inresponse to the control information.

FIG. 11 is a diagram showing an example of the change in the display ofthe virtual object 210 in the display device 26. The virtual object 210moves in the direction of the arrow X in the figure in accordance withthe movement of the worker's finger. When reaching the end of the screen220 in the direction of the arrow X in the figure, a portion of thevirtual object 210 which protrudes from the screen 220 in accordancewith the movement is gradually not displayed. In this manner, thedisplay of the virtual object 210 moves so as to go out of the screen220 along the direction of the arrow X in the figure. The display of thevirtual object 210 on the display device 16 also changes in the sameway.

In step S116, a response generating unit 110 generates responseinformation based on the state recognized in step S111. In this example,the response information of “YES” corresponding to the movement of thefinger moving in the direction of the arrow X in FIG. 9 is generated.

In step S117, the server device 30 transmits the response informationgenerated in step S116 to the remote terminal 20. The remote terminal 20receives the response information from the server device 30.

In step S118, the second display unit 107 displays the responseinformation on the display device 26. The response information isdisplayed at a position corresponding to the instruction information. Inthis example, the response information “YES” is displayed at theposition corresponding to the instruction information “please photographthe right side” in the talk area 232, as shown in FIG. 11.

The instructor can recognize that the worker responds to the instructionof “please photograph the right side” as “YES” by viewing the change inthe display of the virtual object 210 and the display of the responseinformation.

The processing of steps S112 to S115 and the processing of steps S116 toS118 may be performed in parallel. Alternatively, the processing ofsteps S112 to S115 and the processing of steps S116 to S118 may beperformed in reverse order, that is, the processing of steps S112 toS115 may be performed after the processing of steps S116 to S118 isperformed.

According to the above-described embodiments, the instruction of thesecond user using the remote terminal 20 is transmitted to the firstuser using the wearable terminal 10 by using the virtual object 210. Theresponse of the first user to this instruction is transmitted to thesecond user by the change in the display of the virtual object 210 andthe response information. As a result, even when the user using thewearable terminal 10 is in an environment where it is difficult toreceive remote work support by the voice, smooth remote work support canbe realized.

MODIFIED EXAMPLES

The present invention is not limited to the above-described embodimentsand may be variously modified. In addition, the following modifiedexamples may be implemented in combination with each of theabove-described embodiments, or may be implemented in combination withother modified examples.

Modified Example 1

While the virtual object 210 has been exemplified in the above-describedembodiments, a virtual object according to the present invention is notlimited to the virtual object 210, and various modifications may be madeon the content and display format thereof.

FIG. 12 is a diagram showing an example of a screen 240 displayed on adisplay device 16 in this modified example. The screen 240 includes avirtual object 241 in addition to the above-described photographed image200. The virtual object 241 is superimposed and displayed on thephotographed image 200. Although an arrow 212 and response information213 are described in the virtual object 241 as in the above-describedvirtual object 210, the above-described arrow 214 and responseinformation 215 are not described. In this case, if a worker performs agesture for moving the virtual object 241 in an X direction in thefigure, a response “YES” is performed in response to an instruction bythe instructor like the above-described embodiments. On the other hand,when the worker does not perform this gesture and a predetermined timehas elapsed, timeout occurs and a response “NO” is performed in responseto the instruction by the instructor. In this case, a responsegenerating unit 110 generates response information “NO”. Thephotographed image 200 and the virtual object 241 are superimposed anddisplayed in a display area 231 of a display device 26, as in thedisplay device 16. In addition, the response information “NO” isdisplayed in a talk area 232 of the display device 26.

FIG. 13 is a diagram showing another example of a screen 250 displayedon a display device 16 in this modified example. The screen 250 includesvirtual objects 251, 252, and 253 in addition to the above-describedphotographed image 200. The objects 251, 252, and 253 are superimposedand displayed on the photographed image 200. Instruction information“please photograph the right side” is described in the virtual object251, as in the above-described virtual object 210. Response information“YES” is described in the virtual object 252. Response information “NO”is described in the virtual object 253. The virtual objects 252 and 253function as software keys. In this example, a gesture for touching thevirtual object 252 and a response “YES” are associated with each other.Further, a gesture for touching the virtual object 253 and a response“NO” are associated with each other. In other words, the response “YES”is predetermined for the virtual object 252 and the response “NO” ispredetermined for the virtual object 253.

For example, when responding to the instruction of the worker as “YES”,the worker performs the gesture for touching the virtual object 252. Inthis case, an image recognizing unit 108 recognizes this gesture. Morespecifically, the image recognizing unit 108 recognizes a position of afinger having a predetermined relationship with the virtual object 252.This predetermined relationship may be, for example, a positionoverlapping with the virtual object 252 in a field of view of theworker. A response generating unit 110 generates response information“YES” corresponding to the virtual object 252. The photographed image200 and the virtual objects 251, 252, and 253 are superimposed anddisplayed in a display area 231 of a display device 26, as in thedisplay device 16. Further, response information “YES” is displayed in atalk area 232 of the display device 26. In this case, a changing unit109 may change the display of the virtual object 252 displayed on thedisplay device 16 and the display device 26. For example, the changingunit 109 may change the color, shape, or size of the virtual object 252.

Modified Example 2

While a response “YES” or “NO” has been exemplified in theabove-described embodiments, the response according to the presentinvention is not limited to “YES” or “NO”. For example, the response maybe a numerical value or a character. In this case, a virtual object forinputting the numerical value or the character is used. This virtualobject may be generated and stored in a storage device 34 in advance.

FIG. 14 is a diagram showing an example of a screen 260 displayed on adisplay device 16 in this modified example. The screen 260 includes avirtual object 261 in addition to the above-described photographed image200. The virtual object 261 is superimposed and displayed on thephotographed image 200. The virtual object 261 functions as a softwaredial. The virtual object 261 is provided with scales of “1” to “5”. Inthis example, rotation amounts corresponding to these scales areassociated with responses including the numerical value of “1” to “5”,respectively. In other words, the responses including numerical valuesof “1” to “5” are predetermined for the rotation amounts correspondingto the scales of “1” to “5”.

For example, it is assumed that instruction information indicates aninstruction “please tell me the thickness of an object”. Since it isnecessary to input a numerical value in order to respond to thisinstruction, the virtual object 261 used for inputting the numericalvalue is displayed on the display devices 16 and 26. That is, thevirtual object 261 is responsive to the instruction information. In thiscase, the worker measures the thickness of the object using, forexample, a caliper. For example, when the thickness of the object is“3”, the worker performs a gesture for rotating the virtual object 261to the scale “3”. More specifically, the worker rotates and moves afinger clockwise from a position corresponding to an initial position toa position corresponding to the scale “3” in an area having apredetermined relationship with the virtual object 261. The area havingthe predetermined relationship may be, for example, an area along acircumference of the virtual object 261.

In this case, an image recognizing unit 108 recognizes this gesture.More specifically, the image recognizing unit 108 recognizes a movementof the finger that rotates the finger clockwise in the area having thepredetermined relationship with the virtual object 261. In accordancewith the movement of the finger, a changing unit 109 rotates the virtualobject 261 displayed on the display devices 16 and 26 to the scale “3”.The response generating unit 110 generates response information “3”corresponding to the rotation amount of the virtual object 261. Thephotographed image 200 and the virtual object 261 are superimposed anddisplayed in a display area 231 of the display device 26, as in thedisplay device 16. Further, the response information “3” is displayed ina talk area 232 of the display device 26.

In another example, a virtual object may function as a softwarekeyboard. In this case, the virtual object has a plurality of key areasthat correspond to a plurality of characters, respectively. A workerperforms a gesture for touching a key area corresponding to a desiredcharacter. In this case, an image recognizing unit 108 recognizes aposition of a finger having a predetermined relationship with this keyarea. The position having the predetermined relationship may be, forexample, a position overlapping with the key area in a field of view ofthe worker. A response generating unit 110 generates responseinformation including the character corresponding to this key area.

Modified Example 3

In the above-described embodiments, various responses may be performedby using gestures. The response may include a request or a question toan instructor. In this case, a virtual object may include a list ofgestures.

FIG. 15 is a diagram showing an example of a screen 270 displayed on adisplay device 16 in this modified example. The screen 270 includes avirtual object 271 in addition to the above-described photographed image200. The virtual object 271 is superimposed and displayed on thephotographed image 200. A list of gestures is described in the virtualobject 271. For example, the list of gestures may include a plurality ofpieces of gesture information corresponding to a plurality of responsecandidates for an instruction by an instructor. In other words, thevirtual object 271 is responsive to the instruction information. Eachpiece of gesture information includes information for representing agesture with a character or an image and response informationcorresponding to the gesture. A worker selects a gesture correspondingto desired response information from among a plurality of gesturesindicated by the list of gestures and performs the selected gesture. Inthis case, a response generating unit 110 generates response informationcorresponding to this gesture. The photographed image 200 and thevirtual object 271 are superimposed and displayed in a display area 231of the display device 26, as in the display device 16. Further, theresponse information generated by the response generating unit 110 isdisplayed in a talk area 232 of the display device 26.

Modified Example 4

In the above-described embodiments, a three-dimensional virtual objectmay be provided from a remote terminal 20 to a wearable terminal 10.Here, it is assumed that a three-dimensional drawing of a certainbuilding is provided from an instructor to a worker. In this case, theremote terminal 20 transmits a drawing data representing thisthree-dimensional drawing to the wearable terminal 10 directly or via aserver device 30. In this case, the drawing data may be used asinstruction information. Upon receiving this drawing data, the wearableterminal 10 displays the three-dimensional drawing corresponding to thereceived drawing data on a display device 16 as a virtual object 281.

FIG. 16 is a diagram showing an example of a screen 280 displayed on thedisplay device 16 in this modified example. The screen 280 includes thevirtual object 281 in addition to the above-described photographed image200. The virtual object 281 is superimposed and displayed on thephotographed image 200. The virtual object 281 is responsive toinstruction information. The photographed image 200 and the virtualobject 281 are overlapped and displayed in a display area 231 of thedisplay device 26, as in the display device 16. For example, it isassumed that the instruction information indicates an instruction“please tell me where a machine room is.” In this case, the workerperforms a gesture for operating the virtual object 281 with a finger todisplay a location of the machine room, and then performs a gesture forpointing to the location of the machine room. The gesture to be operatedmay be, for example, a gesture that moves or rotates the virtual object281. In this case, a changing unit 109 moves the virtual object 281 inaccordance with the movement of the finger. For example, in a case ofthe gesture for moving or rotating the virtual object 281, the virtualobject 281 is moved or rotated. Further, an image recognizing unit 108recognizes a position of the finger corresponding to the location of themachine room on the virtual object 281. A response generating unit 110generates response information indicating the position of the machineroom on the virtual object 281. In this case, in the virtual object 281displayed in the display area 231 of the display device 26, apredetermined mark may be added to the position indicated by theresponse information.

Modified Example 5

While an example in which instruction information is generated based ona voice of an instructor has been described in the above-describedembodiments, the instruction information may be generated in advance.For example, the instruction information may be a questionnaire to beused for checking work processes. The questionnaire includes a pluralityof items that correspond to a plurality of work processes, respectively.In each item, facts to be checked in the work process are described inquestion form. In this case, a remote terminal 20 sequentially transmitsa plurality of item data indicating a plurality of items included in thequestionnaire to a wearable terminal 10 directly or via a server device30. The transmission of the first item data may be performed accordingto an operation of a worker or may be performed when a predeterminedcondition is satisfied. The predetermined condition may be, for example,a condition that a predetermined time has come. The transmission of thesecond and subsequent item data may be performed when a response to theimmediately preceding item is performed. The wearable terminal 10receives the item data. In this case, the items included in thequestionnaire and the above-described virtual object 261 are displayedon a display device 16 and a display device 26.

FIG. 17 is a diagram showing an example of a screen 290 displayed on thedisplay device 16 in this modified example. The screen 290 includesvirtual objects 291 and 292 in addition to the photographed image 200and the virtual object 261 described above. The virtual objects 261,291, and 292 are superimposed and displayed on the photographed image200. The virtual object 291 includes the first item included in thequestionnaire. Here, it is assumed that the first item is a question“how many times have you checked the work content?” For example, whenchecking the work content twice, a worker performs a gesture forrotating the virtual object 261 to a scale “2”. In this case, an imagerecognizing unit 108 recognizes this gesture as in the above-describedmodified example 2. A changing unit 109 rotates the virtual object 261to the scale “2” in accordance with the movement of the finger. Aresponse generating unit 110 generates response information “2”corresponding to the rotation amount of the virtual object 261. In thiscase, the response information “2” is described in the virtual object292. The photographed image 200, the virtual object 261, and the virtualobjects 291 and 292 are superimposed and displayed in a display area 231of the display device 26, as in the display device 16. Further, theresponse information “2” is displayed in a talk area 232 of the displaydevice 26. In this manner, when the response to the first item isreturned, the second item is described in the virtual object 291. Thisprocess is repeated, for example, until responses to all the itemsincluded in the questionnaire are returned.

Modified Example 6

In the above-described embodiments, one instructor may instruct work toa plurality of workers. In this case, a plurality of wearable terminals10 are provided. The wearable terminals 10 are worn by the plurality ofworkers, respectively.

FIG. 18 is a diagram showing an example of a screen 300 displayed on adisplay device 26 in this modified example. The screen 300 has a displayarea 301 similar to the above-described display area 231 and a talk area302 similar to the above-described talk area 232. The display area 301includes a plurality of photographed images 200, 201, and 202photographed by the plurality of wearable terminals 10. In this case, aresponse generating unit 110 generates a plurality of pieces of responseinformation for the plurality of workers. These pieces of responseinformation are displayed in the talk area 232 of the display device 26.These pieces of response information are displayed in different displayformats. For example, these pieces of response information may havedifferent colors, shapes, or sizes. In addition, worker information on acorresponding worker may be added to each piece of response information.The worker information may be, for example, a face image of the worker.

Modified Example 7

In the above-described embodiments, a composite image of a photographedimage 200 and a virtual object 210 may be generated in a wearableterminal 10, a remote terminal 20, or a server device 30. In this case,the photographed image 200 and the virtual object 210 may be combined ata position and a size for reproducing a field of view of a worker.Further, a size, a position, or an angle of the virtual object 210 maybe changed by an operation of the worker.

Modified Example 8

In the above-described embodiments, an instruction of an instructor maybe inputted by means other than a voice. In this case, a voicerecognizing unit 103 may not be provided. For example, a worker mayinput a character data indicating an instruction using an input device25. In this case, an accepting unit 102 accepts an input of thecharacter data. An object generating unit 104 generates a virtual objectbased on the character data. As another example, when stamp imagesindicating a plurality of instructions are prepared, the worker selectsa stamp image indicating a desired instruction from among the stampimages using the input device 25. In this case, the accepting unit 102accepts an input of an image data indicating the selected stamp image.The object generating unit 104 generates the virtual object based on theimage data. In other words, instruction information for the photographedimage 200 may be a voice data indicating a voice of a second user, ormay be a text data or an image data inputted by the operation of thesecond user.

Modified Example 9

In the above-described embodiments, a target photographed by a camera 17is not limited to a space in front of a first user. The targetphotographed by the camera 17 may be around the first user. Further, avirtual object 210 may be generated and stored in at least one of astorage device 14, a storage device 24, and a storage device 34 inadvance. In this case, an object generating unit 104 may not beprovided. Furthermore, a worker may perform a gesture using a body part,such as a head or a foot, other than a finger.

Modified Example 10

While an example in which a wearable terminal according to the presentinvention is a binocular glasses-type wearable terminal 10 of has beendescribed in the above-described embodiments, the wearable terminal isnot limited to this example. For example, the wearable terminal may be amonocular glass-type wearable terminal or a head mount-type wearableterminal.

Modified Example 11

In the above-described embodiments, programs for realizing functions ofa wearable terminal 10 may be executed by a single processor 11, or maybe simultaneously or sequentially executed by two or more processors 11.The same applies to a remote terminal 20 and a server device 30.

Modified Example 12

A subject for implementing functions of a remote work supporting system1 described in the above-described embodiments is merely an example andis not limited thereto. For example, a wearable terminal 10, a remoteterminal 20, or both may have at least part of functions of the serverdevice 30. For example, the remote terminal 20 may function as the voicerecognizing unit 103, the object generating unit 104, and the screensharing unit 105 described above, and the wearable terminal 10 mayfunction as the screen sharing unit 105, the image recognizing unit 108,the changing unit 109, and the response generating unit 110 describedabove. In this case, the server device 30 may have only a function ofrelaying data between the wearable terminal 10 and the remote terminal20. Alternatively, the server device 30 may not be provided.

Modified Example 13

Steps of processing performed in a remote work supporting system 1 arenot limited to an example described in the above-described embodiments.The steps of processing may be interchanged unless there is acontradiction. Further, the present invention may be provided as aremote work supporting method including the steps of processingperformed in the remote work supporting system 1.

Modified Example 14

The present invention may be provided as programs to be executed in awearable terminal 10, a remote terminal 20, or a server device 30. Theprograms may be downloaded via a network such as the Internet or thelike. Further, the programs may be provided in a state where they arerecorded on a computer-readable recording medium such as a magneticrecording medium (a magnetic tape, a magnetic disk, etc.), an opticalrecording medium (optical disk, etc.), a magneto-optical recordingmedium, a semiconductor memory, or the like.

Modified Example 15

In the above-described embodiments, a virtual object 210 and aphotographed image 200 may be combined in a wearable terminal 10 or aremote terminal 20. If the combining is performed in the wearableterminal 10, instruction information is inputted from the remoteterminal 20 to the wearable terminal 10. The wearable terminal 10generates a composite image into which the virtual object and thephotographed image 200 are combined according to the instructioninformation, and displays the composite image on the display device 16.On the other hand, if the combining is performed in the remote terminal20, the remote terminal 20 generates the composite image into thevirtual object 210 and the photographed image 200 are combined accordingto instruction information inputted by a second user, and transmits acomposite image data representing the composite image to the wearableterminal 10. The wearable terminal 10 displays the composite imagecorresponding to the composite image data received from the remoteterminal 20 on a display device 16. Comparing a case in which thecombining is performed in the wearable terminal 10 and a case in whichthe combining is performed in the remote terminal 20, a data transferspeed increases in the former case since the transmission of thecomposite image data does not occur.

1. A remote work supporting system for sharing an image photographed bya wearable terminal between the wearable terminal and a remote terminalin order to support remote work, comprising: an image acquiring unitthat acquires an image photographed by the wearable terminal; a screensharing unit that screen-shares the image between the wearable terminaland the remote terminal; an accepting unit that accepts an input ofinstruction information for the image from the remote terminal; a firstdisplay unit that superimposes and displays the image and a virtualobject corresponding to the instruction information on the wearableterminal; a second display unit that superimposes and displays the imageand the virtual object on the remote terminal; an image recognizing unitthat analyzes the image to recognize a state of a finger; a changingunit that when a movement of the finger moving from a position having apredetermined relationship with the virtual object is recognized by theimage recognizing unit, changes a display of the virtual object by thefirst display unit and the second display unit to move the virtualobject in accordance with the recognized movement state; and a responsegenerating unit that generates response information that ispredetermined for a moving direction of the virtual object, wherein thesecond display unit further displays the response information generatedby the response generating unit.
 2. The remote work supporting systemaccording to claim 1, wherein the instruction information is a voicedata indicating a voice of a user of the remote terminal, wherein theremote work supporting system further comprises a voice recognizing unitthat recognizes a content of the voice by analyzing the voice data, andwherein the virtual object is generated in accordance with the contentof the recognized voice. 3-4. (canceled)
 5. The remote work supportingsystem according to claim 1, wherein each of the first display unit andthe second display unit displays a plurality of virtual objectscorresponding to the instruction information, and wherein when aposition of the finger having a predetermined relationship with any oneof the plurality of virtual objects is recognized by the imagerecognizing unit, the response generation unit generates the responseinformation that is predetermined for the object.
 6. The remote worksupporting system according to claim 1, wherein when a movement of thefinger rotating and moving in an area having a predeterminedrelationship with the virtual object is recognized by the imagerecognizing unit, the changing unit rotates the virtual object inaccordance with the recognized movement, and wherein the responsegenerating unit generates the response information that is predeterminedfor a rotation amount of the virtual object.
 7. The remote worksupporting system according to claim 1, wherein the virtual objectindicates a list of a plurality of states of the finger, and whereinwhen any one of the plurality of states is recognized by the imagerecognizing unit, the response generating unit generates the responseinformation that is predetermined for the recognized state.
 8. Theremote work supporting system according to claim 1, wherein a pluralityof users use a plurality of wearable terminals, respectively, whereinthe first display unit is provided for each of the plurality of wearableterminals, wherein the response generating unit generates a plurality ofpieces of response information for the plurality of users respectively,and wherein the second display unit displays the plurality of pieces ofresponse information in different display formats.
 9. The remote worksupporting system according to claim 1, wherein the virtual object is athree-dimensional virtual object, wherein when a movement of the fingeroperating the virtual object is recognized by the image recognizingunit, the changing unit changes the virtual object in accordance withthe recognized movement.
 10. The remote work supporting system accordingto claim 1, wherein the wearable terminal is worn on a head of a user,wherein the first display unit displays the image and the virtual objecton a display panel having light transparency provided within a field ofview of the user, and wherein the user performs an operation on thevirtual object using the finger seen through the display panel.
 11. Aremote work supporting method for screen-sharing an image photographedby a wearable terminal between the wearable terminal and a remoteterminal in order to support remote work, comprising: an image acquiringstep of acquiring an image photographed by the wearable terminal; ascreen sharing step of screen-sharing the image between the wearableterminal and the remote terminal; an accepting step of accepting aninput of instruction information for the image from the remote terminal;a first display step of superimposing and displaying the image and avirtual object corresponding to the instruction information on thewearable terminal; a second display step of superimposing and displayingthe image and the virtual object on the remote terminal, an imagerecognizing step of analyzing the image to recognize a state of afinger; a changing step of, when a movement of the finger moving from aposition having a predetermined relationship with the virtual object isrecognized, changing a display of the virtual object by the firstdisplay step and the second display step to move the virtual object inaccordance with the recognized movement; a response generating step ofgenerating response information that is predetermined for a movingdirection of the virtual object, and a third display step of displayingthe generated response information on the remote terminal.
 12. A programfor screen-sharing an image photographed by a wearable terminal betweenthe wearable terminal and a remote terminal in order to support remotework, the program causing a computer to execute: an image acquiring stepof acquiring an image photographed by the wearable terminal; a screensharing step of screen-sharing the image between the wearable terminaland the remote terminal; an accepting step of accepting an input ofinstruction information for the image from the remote terminal; a firstdisplay step of superimposing and displaying the image and a virtualobject corresponding to the instruction information on the wearableterminal; a second display step of superimposing and displaying theimage and the virtual object on the remote terminal, an imagerecognizing step of analyzing the image to recognize a state of afinger; a changing step of, when a movement of the finger moving from aposition having a predetermined relationship with the virtual object isrecognized, changing a display of the virtual object by the firstdisplay step and the second display step to move the virtual object inaccordance with the recognized movement; a response generating step ofgenerating response information that is predetermined for a movingdirection of the virtual object, and a third display step of displayingthe generated response information on the remote terminal.
 13. A remotework supporting system for sharing images photographed by a plurality ofwearable terminals being respectively used by a plurality of usersbetween each of the wearable terminals and a remote terminal in order tosupport remote work, comprising: an image acquiring unit that acquiresan image photographed by each of the wearable terminals; a screensharing unit that screen-shares the image between each of the wearableterminals and the remote terminal; an accepting unit that accepts aninput of instruction information for the image from the remote terminal;a first display unit that is provided for each of the wearableterminals, and superimposes and displays the image and a virtual objectcorresponding to the instruction information on each of the wearableterminals; a second display unit that superimposes and displays theimage and the virtual object on the remote terminal; an imagerecognizing unit that analyzes the image to recognize a state of a bodypart; a changing unit that changes a display of the virtual object bythe first display unit and the second display unit based on therecognized state; and a response generating unit that generates aplurality of pieces of response information for the plurality of usersrespectively, based on the state recognized by the image recognizingunit and the display of the virtual object, wherein the second displayunit displays the plurality of pieces of response information indifferent display formats.
 14. A remote work supporting method forsharing images photographed by a plurality of wearable terminals beingrespectively used by a plurality of users between each of the wearableterminals and a remote terminal in order to support remote work,comprising: an image acquiring step of acquiring an image photographedby each of the wearable terminals; a screen sharing step ofscreen-sharing the image between each of the wearable terminals and theremote terminal; an accepting step of accepting an input of instructioninformation for the image from the remote terminal; a first display stepof superimposing and displaying the image and a virtual objectcorresponding to the instruction information on each of the wearableterminals; a second display step of superimposing and displaying theimage and the virtual object on the remote terminal; an imagerecognizing step of analyzing the image to recognize a state of a bodypart; a changing step of changing a display of the virtual object by thefirst display step and the second display step based on the recognizedstate; a response generating step of generating a plurality of pieces ofresponse information for the plurality of users respectively, based onthe recognized state and the display of the virtual object; and a thirddisplay step of displaying on the remoter terminal the plurality ofpieces of response information in different display formats.
 15. Aprogram for sharing images photographed by a plurality of wearableterminals being respectively used by a plurality of users between eachof the wearable terminals and a remote terminal in order to supportremote work, the program causing a computer to execute: an imageacquiring step of acquiring an image photographed by each of thewearable terminals; a screen sharing step of screen-sharing the imagebetween each of the wearable terminals and the remote terminal; anaccepting step of accepting an input of instruction information for theimage from the remote terminal; a first display step of superimposingand displaying the image and a virtual object corresponding to theinstruction information on each of the wearable terminals; a seconddisplay step of superimposing and displaying the image and the virtualobject on the remote terminal; an image recognizing step of analyzingthe image to recognize a state of a body part; a changing step ofchanging a display of the virtual object by the first display step andthe second display step based on the recognized state; a responsegenerating step of generating a plurality of pieces of responseinformation for the plurality of users respectively, based on therecognized state and the display of the virtual object; and a thirddisplay step of displaying on the remoter terminal the plurality ofpieces of response information in different display formats.